Stereo assist with rolling shutters

ABSTRACT

An imaging system for a vehicle may include a first image capture device having a first field of view and configured to acquire a first image relative to a scene associated with the vehicle, the first image being acquired as a first series of image scan lines captured using a rolling shutter. The imaging system may also include a second image capture device having a second field of view different from the first field of view and that at least partially overlaps the first field of view, the second image capture device being configured to acquire a second image relative to the scene associated with the vehicle, the second image being acquired as a second series of image scan lines captured using a rolling shutter. As a result of overlap between the first field of view and the second field of view, a first overlap portion of the first image corresponds with a second overlap portion of the second image. The first image capture device has a first scan rate associated with acquisition of the first series of image scan lines that is different from a second scan rate associated with acquisition of the second series of image scan lines, such that the first image capture device acquires the first overlap portion of the first image over a period of time during which the second overlap portion of the second image is acquired.

This application is a continuation of and claims priority to U.S. patentapplication Ser. No. 15/374,748, filed Dec. 9, 2016, (now U.S. Pat. No.9,854,185), which is a continuation of and claims priority to U.S.patent application Ser. No. 15/045,823, filed Feb. 17, 2016 (now U.S.Pat. No. 9,531,966), which is a continuation of and claims priority toU.S. patent application Ser. No. 14/534,750, filed Nov. 6, 2014, (nowU.S. Pat. No. 9,286,522), which is a continuation of and claims priorityto U.S. patent application Ser. No. 14/156,096, filed Jan. 15, 2014 (nowU.S. Pat. No. 8,908,041), which claims priority under 35 U.S.C. § 119 toU.S. Provisional Application No. 61/752,515, filed Jan. 15, 2013, andU.S. Provisional Application No. 61/761,724, filed Feb. 7, 2013. Eachaforementioned application is incorporated herein by reference in itsentirety.

BACKGROUND I. Technical Field

The present disclosure relates generally to camera imaging systems andmore specifically to devices and techniques for capturing images in arolling shutter, stereo image acquisition system that may be included ona vehicle.

II. Background Information

Camera based driver assistance systems for use in vehicles may includemonocular object detection systems that rely primarily on a singlecamera to collect images. Because the images in these types of systemsare captured from a single point of view, direct determination ofdistance to a target object can be challenging. Therefore, monocularobject detection systems may rely upon estimation techniques toindirectly determine distances to a target object based on informationabout the object class and/or contextual information relative to thecontext of the object (e.g., aspects of a road plane on which the targetobject resides). In some cases, monocular systems may use patternrecognition to detect a specific object class prior to monocular rangeestimation.

Camera based driver assistance systems for use in vehicles may alsoinclude stereo systems that employ two cameras. In some systems, thesecameras may be mounted side-by-side where epipolar lines are alignedwith the horizontal image scan lines. Such a system may use a densedisparity map to create a 3D map of the environment. The system may thenuse this 3D representation for foreground and/or backgroundsegmentation, for instance, to find candidate regions for furtherprocessing. The system may also use the 3D representation to locateinteresting objects or to estimate range and/or range-rate to detectedobjects. Such stereo systems may work well with close and medium rangetargets and in good weather, and may give depth maps on general targets.However, such stereo systems may experience difficulties during adverseweather conditions or where cluttered scenes exist. Additionally, thesesystems may have difficulty imaging objects at longer distances from thevehicle.

Some imaging systems, such as systems described in U.S. Pat. No.7,786,898, may fuse information from both a monocular system and astereo system. This type of system may include a primary cameraresponsible for target detection/selection and range estimation. Asecondary camera may provide stereo-range on selected targets forpurposes of target verification.

Some stereo systems may include an asymmetric configuration that maycombine stereo-depth and monocular depth together. For instance, twoasymmetric cameras (e.g., with different fields of view (FOV) and focallengths) may be employed for independent applications. Additionally,image information from these cameras may be combined to provide stereodepth. For cameras with global shutters, such stereo processing mayinvolve, among other things, cropping the wider FOV camera, smoothingand subsampling of images, and/or rectification in order to provide amatching image pair.

Recent generations of image sensors, including those that may be used inautomotive sensors, may include a rolling shutter. Such a rollingshutter may introduce complications in stereo image processing,especially in asymmetric stereo applications that use cameras havingdifferent fields of view. For instance, if both a wide FOV camera and anarrow FOV camera are aimed at a common scene, then the narrow FOVcamera may overlap with only a portion of the FOV of the wide FOVcamera. If both cameras acquire images as a similar number of image scanlines acquired at a similar line scan rate, then the acquired image scanlines in the area of the overlap in the fields of view of the twocameras will lack synchronization. Such a lack of synchronization mayintroduce difficulties in determining a correspondence of image pointsin a first image from the wide FOV camera with image points in a secondimage from the narrow FOV camera, which can lead to significantinaccuracies object distance measurements.

SUMMARY

Consistent with disclosed embodiments, an imaging system for a vehicleis provided, the system comprising a first image capture device having afirst field of view and configured to acquire a first image relative toa scene associated with the vehicle, the first image being acquired as afirst series of image scan lines captured using a rolling shutter. Theimaging system may also include a second image capture device having asecond field of view different from the first field of view and that atleast partially overlaps the first field of view, the second imagecapture device being configured to acquire a second image relative tothe scene associated with the vehicle, the second image being acquiredas a second series of image scan lines captured using a rolling shutter.As a result of overlap between the first field of view and the secondfield of view, a first overlap portion of the first image may correspondwith a second overlap portion of the second image. The first imagecapture device may have a first scan rate associated with acquisition ofthe first series of image scan lines that may be different from a secondscan rate associated with acquisition of the second series of image scanlines, such that the first image capture device acquires the firstoverlap portion of the first image over a period of time during whichthe second overlap portion of the second image is acquired.

Consistent with disclosed embodiments, a vehicle is disclosed, thevehicle including a body and an imaging system for a vehicle, the systemcomprising a first image capture device having a first field of view andconfigured to acquire a first image relative to a scene associated withthe vehicle, the first image being acquired as a first series of imagescan lines captured using a rolling shutter. The imaging system may alsoinclude a second image capture device having a second field of viewdifferent from the first field of view and that at least partiallyoverlaps the first field of view, the second image capture device beingconfigured to acquire a second image relative to the scene associatedwith the vehicle, the second image being acquired as a second series ofimage scan lines captured using a rolling shutter. As a result ofoverlap between the first field of view and the second field of view, afirst overlap portion of the first image may correspond with a secondoverlap portion of the second image. The first image capture device mayhave a first scan rate associated with acquisition of the first seriesof image scan lines that may be different from a second scan rateassociated with acquisition of the second series of image scan lines,such that the first image capture device acquires the first overlapportion of the first image over a period of time during which the secondoverlap portion of the second image is acquired, wherein the period oftime is associated with a ratio between the first scan rate and thesecond scan rate.

Consistent with disclosed embodiments, an imaging system for a vehicleis provided, the system comprising a first image capture device having afirst field of view and configured to acquire a first image relative toa scene associated with the vehicle, the first image being acquired as afirst series of image scan lines captured using a rolling shutter. Thesystem may also include a second image capture device having a secondfield of view different from the first field of view and that at leastpartially overlaps the first field of view, the second image capturedevice being configured to acquire a second image relative to the sceneassociated with the vehicle, the second image being acquired as a secondseries of image scan lines captured using a rolling shutter. As a resultof overlap between the first field of view and the second field of view,a first overlap portion of the first image may correspond to a secondoverlap portion of the second image. They system may include at leastone processing device configured to: receive the first image from thefirst image capture device; receive the second image from the secondimage capture device; and correlate at least a first area of the firstoverlap portion of the first image with a corresponding second area ofthe second overlap portion of the second image.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this disclosure, illustrate various disclosed embodiments. Inthe drawings:

FIG. 1a is a diagrammatic, side view representation of an exemplaryvehicle imaging system consistent with the presently disclosedembodiments;

FIG. 1b is a diagrammatic, top view illustration of the embodiment shownin FIG. 1 a;

FIG. 1c is a diagrammatic, top view illustration of an exemplary vehicleimaging system with another camera configuration consistent with thepresently disclosed embodiments;

FIG. 2a represents overlapping fields of view of two cameras havingdifferent fields of view, according to an exemplary disclosedembodiment;

FIG. 2b represents overlapping fields of view of two cameras havingdifferent fields of view, according to another exemplary disclosedembodiment;

FIG. 3 represents an exemplary process for use in systems employingcameras with global shutters;

FIG. 4 is an example of displacement of points in the image from thesynchronized line for targets at various distances from the vehicle;

FIG. 5 is an example of depth estimates for a target at a given distancefrom a vehicle;

FIG. 6 shows displacement in y for various distance values after a fixinvolving pre-processing an image with the homography of a target plane;

FIG. 7 shows exemplary process for use in systems to perform depthmeasurements for upright objects;

FIG. 8a shows depth estimates for a target at a given distance from thevehicle for a given vehicle forward motion and lateral velocity;

FIG. 8b shows corrected depth estimates for forward vehicle motion andno lateral velocity;

FIG. 9 is an example of the disparity expected on the road surface dueto the camera baseline;

FIG. 10 is an example of the shift in the y direction as a function ofthe image row;

FIG. 11 shows exemplary process for use in systems to estimate distancesto road features;

FIG. 12a is an example of the disparity error for different rows in theimage introduced by the vehicle forward motion and the time delay;

FIG. 12b is an example of the disparity error as a fraction of thedisparity on that row;

FIG. 13 is an example of the expected y shift due to time delay andforward motion if a given row is synchronized;

FIG. 14a is an example of the ratio of disparity error due to motionover true disparity due to baseline as a function of row (y);

FIG. 14b is an example of the ratio of disparity error due to motionover true disparity due to baseline for the case where the object ismoving laterally;

FIG. 15 shows an exemplary process for use in systems where there ismotion disparity.

DETAILED DESCRIPTION

The following detailed description refers to the accompanying drawings.Wherever possible, the same reference numbers are used in the drawingsand the following description to refer to the same or similar parts.While several illustrative embodiments are described herein,modifications, adaptations and other implementations are possible. Forexample, substitutions, additions or odifications may be made to thecomponents illustrated in the drawings, and the illustrative methodsdescribed herein may be modified by substituting, reordering, removing,or adding steps to the disclosed methods. Accordingly, the followingdetailed description is not limiting of the disclosed embodiments.Instead, the proper scope is defined by the appended claims.

Referring to the accompanying drawings, FIG. 1a is a diagrammatic, sideview representation of an exemplary vehicle imaging system consistentwith the presently disclosed. FIG. 1b is diagrammatic, top viewillustration of the embodiment shown in FIG. 1a . As illustrated in FIG.1a , a disclosed embodiment of the present invention may include avehicle 150 having a system 100 with a first image capture device 110and a second image capture device 120 and a processor 130. While twoimage capture devices 110 and 120 are shown, it should be understoodthat other embodiments may include more than two image capture devices.

It is to be understood that the disclosed embodiments are not limited tovehicles and could be applied in other contexts. It is also to beunderstood that disclosed embodiments are not limited to a particulartype of vehicle 150 and may be applicable to all types of vehiclesincluding automobiles, truck trailers and other types of vehicles.

The processor 130 may comprise various types of devices. For example,processor 130 may include a controller unit, an image preprocessor, acentral processing unit (CPU), support circuits, digital signalprocessors, integrated circuits, memory, or any other types of devicesfor image processing and analysis. The image preprocessor may include avideo processor for capturing, digitizing and processing the imageryfrom the image sensors. The CPU may comprise any number ofmicrocontrollers or microprocessors. The support circuits may be anynumber of circuits generally well known in the art, including cache,power supply, clock and input-output circuits. The memory may storesoftware that when executed by the processor, controls the operation ofthe system. The memory may include databases and image processingsoftware. The memory may comprise any number of random access memory,read only memory, flash memory, disk drives, optical storage, tapestorage, removable storage and other types of storage. In one instance,the memory may be separate from the processor 130. In another instance,the memory may be integrated into the processor 130.

The first image capture device 110 may include any suitable type ofimage capture device. Image capture device 110 may include an opticalaxis 116. In one instance, the image capture device 110 may include anAptina M9V024 WVGA sensor with a global shutter. In other embodiments,image capture device 110 may include a rolling shutter. Image capturedevice 110 may include various optical elements. In some embodiments oneor more lenses may be included, for example, to provide a desired focallength and field of view for the image capture device. In someembodiments, image capture device 110 may be associated with a 6 mm lensor a 12 mm lens. In some embodiments, image capture device may beconfigured to capture images having a desired FOV, including, forexample, a wide FOV, such as a 46 degree FOV, 50 degree FOV, 52 degreeFOV, or greater. In some embodiments, image capture device 110 mayinclude a wide angle bumper camera or one with up to a 180 degree FOV.

The first image capture device 110 may acquire a plurality of firstimages relative to a scene associated with the vehicle 150. Each of theplurality of first images may be acquired as a series of image scanlines, which may be captured using a rolling shutter. Each scan line mayinclude a plurality of pixels.

The first image capture device 110 may have a scan rate associated withacquisition of each of the first series of image scan lines. The scanrate may refer to a rate at which an image sensor can acquire image dataassociated with each pixel included in a particular scan line.

The first image capture device 110 may contain any suitable type ofimage sensor, including CCD sensors or CMOS sensors, for example. In oneembodiment, a CMOS image sensor may be employed along with a rollingshutter, such that each pixel in a row is read one at a time, andscanning of the rows proceeds on a row-by-row basis until an entireimage frame has been captured. In some embodiments, the rows may becaptured sequentially from top to bottom relative to the frame.

The use of a rolling shutter may result in pixels in different rowsbeing exposed and captured at different times, which may cause skew andother image artifacts in the captured image frame. On the other hand,when the image capture device 110 is configured to operate with a globalor synchronous shutter, all of the pixels may be exposed for the sameamount of time and during a common exposure period. As a result, theimage data in a frame collected from a system employing a global shutterrepresents a snapshot of the entire FOV at a particular time. Incontrast, in a rolling shutter application, each row in a frame isexposed and data is capture at different times. Thus, moving objects mayappear distorted in an image capture device having a rolling shutter.This phenomenon will be described in greater detail below.

The second image capture device 120 may be any type of image capturedevice. Like the first image capture device 110, image capture device120 may include an optical axis 126. In one embodiment, image capturedevice 120 may include an Aptina M9V024 VVVGA sensor with a globalshutter. Alternatively, image capture device 120 may include a rollingshutter. Like image capture device 110, image capture device 120 may beconfigured to include various lenses and optical elements. In someembodiments, lenses associated with image capture device 120 may providea FOV that is the same as or narrower than a FOV associated with imagecapture device 110. For example, image capture device 120 may have anFOV of 40 degrees, 30 degrees, 26 degrees, 23 degrees, 20 degrees, orless.

Image capture device 120 may acquire a plurality of second imagesrelative to a scene associated with the vehicle 150. Each of theplurality of second images may be acquired as a second series of imagescan lines, which may be captured using a rolling shutter. Each scanline or row may have a plurality of pixels. The second image capturedevice 120 may have a second scan rate associated with acquisition ofeach of image scan lines included in the second series.

The second image capture device 120 may contain any type of imagesensors, including CCD sensors or CMOS sensors, for example. In oneembodiment, the CMOS image sensor may be associated with a rollingshutter, where the image rows may be exposed and captured sequentiallyone after another to provide each image frame.

Each image capture device 110, 120 may be positioned at any suitableposition and orientation relative to vehicle 150. The relativepositioning of the two image capture devices 110 and 120 may be selectedto aid in fusing together the information acquired from the imagecapture devices. For example, in some embodiments, a FOV associated withimage capture device 120 may overlap partially or fully with a FOVassociated with image capture device 110.

Image capture devices 110 and 120 may be located on vehicle 150 at anysuitable relative heights. In one instance, there may be a heightdifference between the two image capture devices 110 and 120, which mayprovide sufficient parallax information to enable stereo analysis. Thedifference in the height at which the two image capture devices 110 and120 are mounted is denoted by d_(h), as shown in FIG. 1a . There mayalso be a lateral displacement difference between the image capturedevices 110 and 120, giving additional parallax information for stereoanalysis by processing unit 130, for example. The difference in thelateral displacement may be denoted by d_(x), as shown in FIG. 1b . Insome embodiments, there may exist fore or aft displacement (e.g., rangedisplacement) between image capture device 110 and image capture device120. For example, image capture device 110 may be located 0.5, to 2meters or more behind image capture device 120. This type ofdisplacement may enable one of the image capture devices to coverpotential blind spots of the other image capture device(s). Thedifference in the range displacement is denoted by d_(z), as shown inFIG. 1(a). In other embodiments as illustrated in FIG. 1c , there may belateral displacement dx, but range displacement dz=0 and heightdisplacement dh=0.

Image capture device 110 may have any suitable resolution capability(e.g., number of pixels associated with the image sensor), and theresolution of the image sensor(s) associated with image capture device110 may be higher, lower, or the same as the resolution of the imagesensor(s) associated with image capture device 120. In some embodiments,the image sensor(s) associated with image capture device 110 and/orimage capture device 120 may have a resolution of 640×480, 1024×768,1280×960, or any other suitable resolution.

The frame rate (e.g., the rate at which an image capture device acquiresa set of pixel data of one image frame before moving on to capture pixeldata associated with the next image frame) may be controllable. Theframe rate associated with image capture device 110 may be higher,lower, or the same as the frame rate associated with image capturedevice 120. The frame rate associated with the image capture devices 110and 120 may depend on a variety of factors that may affect the timing ofthe frame rate. For example, one or both of image capture devices 110and/or 120 may include a selectable pixel delay period imposed before orafter acquisition of image data associated with one or more pixels of animage sensor in image capture device 110 and/or 120. Generally, imagedata corresponding to each pixel may be acquired according to a clockrate for the device (e.g., one pixel per clock cycle). Additionally, inembodiments including a rolling shutter, one or both of image capturedevices 110 and/or 120 may include a selectable horizontal blankingperiod imposed before or after acquisition of image data associated witha row of pixels of an image sensor in image capture device 110 and/or120. Further, one or both of image capture devices 110 and/or 120 mayinclude a selectable vertical blanking period imposed before or afteracquisition of image data associated with an image frame of imagecapture device 110 and/or 120.

These timing controls may enable synchronization of frame ratesassociated with image capture device 110 and 120, even where the linescan rates of each are different. And, as will be discussed in greaterdetail below, these selectable timing controls, among other factors(e.g., image sensor resolution, maximum line scan rates, etc.) mayenable synchronization of image capture from an area where the field ofview of image capture device 110 overlaps with a field of view of imagecapture device 120, even where the field of view of image capture device110 is different from the field of view of image capture device 120.

Additionally, frame rate timing in image capture device 110 and/or 120may depend on the resolution of the associated image sensors. Forexample, assuming similar line scan rates for both devices, if onedevice includes an image sensor having a resolution of 640×480 andanother device includes an image sensor with a resolution of 1280×960,then more time will be required to acquire a frame of image data fromthe sensor having the higher resolution.

Another factor that may affect the timing of image data acquisition inimage capture device 110 and/or 120 is the maximum line scan rate. Forexample, acquisition of a row of image data from an image sensorincluded in image capture device 110 and/or 120 will require someminimum amount of time. Assuming no pixel delay periods are added, thisminimum amount of time for acquisition of a row of image data will berelated to the maximum line scan rate for a particular device. Devicesthat offer higher maximum line scan rates have the potential to providehigher frame rates than devices with lower maximum line scan rates. Insome embodiments, image capture device 120 may have a maximum line scanrate that is higher than a maximum line scan rate associated with imagecapture device 110. In some embodiments, the maximum line scan rate ofimage capture device 120 may be 1.25, 1.5, 1.75, or 2 times or moregreater than a maximum line scan rate of image capture device 110.

In a another embodiment, image capture devices 110 and 120 may have thesame maximum line scan rate, but image capture device 110 may beoperated at a scan rate less than or equal to its maximum scan rate. Thesystem may be configured such that image capture device 120 operates ata line scan rate that is equal to the line scan rate of image capturedevice 110. In other instances, the system may be configured such thatthe line scan rate of image capture device 120 may be 1.25, 1.5, 1.75,or 2 times or more greater than the line scan rate of image capturedevice 110.

In some embodiments, image capture devices 110 and 120 may beasymmetric. That is, they may include cameras having different fields ofview (FOV) and focal lengths. The fields of view of image capturedevices 110 and 120 may include any desired area relative to anenvironment of vehicle 150, for example. In some embodiments, either orboth of image capture devices 110 and 120 may be configured to acquireimage data from an environment in front of vehicle 150, behind vehicle150, to the sides of vehicle 150, or combinations thereof.

Further, the focal length associated with each image capture device 110and/or 120 may be selectable (e.g., by inclusion of appropriate lensesetc.) such that each device acquires images of objects at a desireddistance range relative to vehicle 150. For example, in some embodimentsimage capture devices 110 and 120 may acquire images of close-up objectswithin a few meters from the vehicle. Image capture devices 110 and 120may also be configured to acquire images of objects at ranges moredistant from the vehicle (e.g., 25 m, 50 m, 100 m, 150 m, or more).Further, the focal lengths of image capture devices 110 and 120 may beselected such that one image capture device (e.g., image capture device110) can acquire images of objects relatively close to the vehicle(e.g., within 10 m or within 20 m) while the other image capture device(e.g., image capture device 120) can acquire images of more distantobjects (e.g., greater than 20 m, 50 m, 100 m, 150 m, etc.) from vehicle150.

The field of view associated with each of image capture devices 110 and120 may depend on the respective focal length. For example, as the focallength increases, the corresponding field of view decreases.

Image capture device 110 and image capture device 120 may be configuredto have any suitable field of view. In one particular example, imagecapture device 110 may have a horizontal FOV of 46 degrees and imagecapture device 120 may have a horizontal FOV of 23 degrees. In anotherinstance, image capture device 110 may have a horizontal FOV of 52degrees and image capture device 120 may have a horizontal FOV of 26degrees. In some embodiments, a ratio of the FOV of image capture device110 to the FOV of image capture device 120 may vary between 1.5 to 2.0.In other embodiments, this ratio may vary between 1.25 and 2.25.

System 100 may be configured so that a field of view of image capturedevice 110 overlaps with a field of view of image capture device 120 (atleast partially or fully). In some embodiments, system 100 may beconfigured such that the field of view of image capture device 120, forexample, falls within (e.g., is narrower than) and shares a commoncenter with the field of view of image capture device 110. In otherembodiments, the image capture devices 110 and 120 may capture adjacentFOVs or may have partial overlap in their FOVs. In some embodiments, thefields of view of devices 110 and 120 may be aligned such that a centerof the narrower FOV device 120 may be located in a lower half of thefield of view of the wider FOV device 110 (e.g., in the area below line280 in FIG. 2a ).

FIG. 2a is shows overlapping fields of view of two cameras havingdifferent fields of view, according to an exemplary disclosedembodiment. FIG. 2b represents overlapping fields of view of two camerashaving different fields of view, according to another exemplarydisclosed embodiment.

The wide FOV camera 110 has a first FOV 250, which may be defined by ahorizontal FOV 210 and a vertical FOV 220. In one instance, the wide FOVcamera 110 may have a horizontal FOV 210 of 46 degrees and an imageresolution of 1280×960 pixels. The wide FOV camera 110 may have arolling shutter with a rolling scan direction 251. The rolling shutterdirection 251 could be in any direction vertically or horizontally. Inthe embodiment of FIG. 2a , use of the rolling shutter would enablecapture of an image as a plurality of scan lines, e.g., scan lines 252and 253.

The narrow FOV camera 120 has a second FOV 260, which may be defined bya horizontal FOV 230 and a vertical FOV 240. The narrow FOV camera 120may also have a rolling shutter with a rolling scan direction 261. Inone instance, the narrow FOV camera 120 may have a horizontal FOV 230 of23 degrees and an image resolution the same as or different from wideFOV camera 110. For example, in some embodiments, narrow FOV camera 120may have an image resolution of 1280×960. In other embodiments, thenarrow FOV camera 120 may have an image resolution different from wideFOV camera 110 (e.g., 640×480 pixels). The rolling shutter direction 261could be in any direction vertically or horizontally, and the rollingshutter would enable image capture as a plurality of scan lines,including lines 262 and 263, for example.

The narrow FOV 260 and the wide FOV 250 could be setup to overlappartially or fully in FOV. Overlap region 270, shown in cross hatchingin FIGS. 2a and 2b , represents a region where the narrow field of view260 overlaps with the wide field of view 250. As shown in FIG. 2a ,region 270 may correspond to the entire field of view of the narrowimage capture device 120. Further, the fields of view of image capturedevice 110 and 120 may be aligned such that region 270 is centeredwithin the wider field of view of image capture device 110.Alternatively, as shown in FIG. 2b , region 270 need not represent theentire field of view of the narrower field of view image capture device120, nor must it be centered within the field of view of the wider fieldof view image capture device. Rather, as shown in FIG. 2b , region 270may be offset from the center of image capture device 110. Further, thefields of view of image capture device 110 and 120 may be aligned suchthat region 270 corresponds to an overlap region that is less than theentire fields of view of image capture device 110 and image capturedevice 120.

In embodiments that use a rolling shutter, an image frame from imagecapture device 110 may be acquired as a series of sequentially acquiredscan lines, such as lines 252 and 253 in FIG. 2a . Similarly, a rollingshutter associated with image capture device 120 will enable capturingof image frames as a series of sequentially acquired scan lines 263 and263, as shown in FIG. 2a . When image capture device 110 has a focallength and field of view different from the focal length and field ofview of image capture device 120, and rolling shutters are employed onboth devices, image capture device 110 and 120 may capture portions oftheir respective fields of view corresponding to overlap region 270 atdifferent times. Such a lack of synchronization can lead to difficultiesin correlated stereo analysis of the acquired images.

To illustrate further, one exemplary system may include image capturedevices 110 and 120 each having a sensor resolution of 1280×960. Thus,if rolling shutters are employed to acquire image frames from both imagedevices, then the image frames from both image capture devices will beacquired as a series of 960 scan lines each including 1280 pixels ofimage data. In this example, image capture device 110 may have a 52degree field of view, and image capture device may have a 26 degreefield of view, and both fields of view may share a common center, asshown in FIG. 2a . If both image capture devices included a similar linescan rates, then it becomes apparent that the some or all of theportions of the image frames, from each mage capture device, thatcorrespond to overlap region 270 will be acquired at different times.

For example, assuming that both image capture devices begin to acquiretheir respective image frames at the same time (i.e., the first scanline of both image capture devices is synched), then while image capturedevice 120 is acquiring the first line from overlapping region 270,image capture device is acquiring an image line from the top of itsfield of view, for example, which falls outside of overlap region 270.In the example where image capture device 120 has a field of view thatis one-half the angular width of the field of view of image capturedevice 110, image capture device 110 will not acquire an image linecorresponding to overlap region 270 until it reaches scan line 240 (onequarter of the way through its image frame). Again, assuming both imagecapture devices have the same line scan rate, then when image capturedevice 110 reaches its scan line number 240, then image capture device120 will also be acquiring its scan line 240, which is 25% of the waythrough overlap region 270. Further, when image capture device 120finishes the last line of its scan (i.e., scan line 960) it will be atthe bottom of overlap region 270. At the same time, however, imagecapture device 110 will be acquiring the last line of its field of view,which falls outside of overlap region 270. Indeed, only scan lines 240through 720 in image capture device 110 will include overlap region 270,while all 960 scan lines from image capture device 120 will correspondto overlap region 270. Further, in this example, only the center scanline (corresponding to dashed line 280 in FIG. 2a ) will be acquired byboth image capture devices at the same time. All other linescorresponding to overlap region 270 will be acquired at different times,which again, can lead to difficulties in performing stereo analysisassociated with the overlapping region 270.

By adjusting the image acquisition timing control parameters of eachimage capture device, however, it may be possible to ensure that theportions of the image frames of each image capture device correspondingto overlap region 270 are acquired during the same period of time.

In general, this synchronization may be accomplished in a variety ofways. For example, in one embodiment, image capture device 120 (havingthe narrower field of view) may be configured to have a line scan ratethat is different from a line scan rate associated with image capturedevice 110. For example, the line scan rate of image capture device 120may be higher than a line scan rate for image capture device 110. Insome embodiments, the line scan rate for image capture device 120 may betwo times (or more) higher than a line scan rate for image capturedevice 110. Generally, the narrower field of view image capture device120 should have a scan rate at least high enough such that the portionsof the image frames from both image capture devices 110 and 120 thatcorrespond to overlap region 270 may be acquired during the same timeperiod.

Returning to the example in which image capture device 110 has a 52degree field of view and image capture device 110 has a 26 degree fieldof view, and both include image sensor resolutions of 1280×960, overlapregion 270 will correspond to scan lines 240 through 720 (480 scan linestotal) in the image frame of image capture device 110. On the otherhand, overlap region 270 will correspond to the full 960 scan lines inimage capture device 120. As a result, in order for lines 240 through720 in image capture device 110 to be captured during the same timeperiod as the 960 scan lines of image capture device 120, the line scanrate of image capture device 120 could be at least two times higher thana line scan rate of image capture device 110. Such an arrangement mayenable image capture device 120 to capture all 960 lines in the sameamount of time that it takes image capture device 110 to acquire the 480scan lines from lines 240 to 720 that correspond to overlap region 270.

Line scan rate, however, is not the only timing control parameter thatcan enable synching of the acquisition of those portions of therespective image frames corresponding to overlap region 270. Thehorizontal blanking period, vertical blanking period, and pixel delayfor one or both of image capture devices 110 and 120 may be selected toachieve the desired capture synch in the overlap region 270. Forexample, in embodiments where the line scan rate of image capture device120 may be too low to ensure that acquisition of the images from overlapregion 270 are synched, the horizontal blanking period of image capturedevice 110 (the wider field of view device) may be increased to slow theeffective line scan rate of image capture device 110. Additionally oralternatively, a pixel delay associated with image capture device 110may be increased to slow the effective line scan rate of image capturedevice 110.

Even where image capture device 120 has a sufficient line scan rate toenable synching of image acquisition in overlap region 270, the totalframe rate of each image capture device may be the same so that thesynching of image acquisition in overlap region 270 is repeatable.Returning to the example, to synch image acquisition in overlap region270, image capture device 120 should begin acquiring data at scan linenumber 1 when image capture device 110 reaches scan line 240. Whileimage capture device 110 acquires lines 240 to 720, image capture device120 acquires all 960 of its scan lines. While image capture device 110is capturing lines 721 to 960 and 1 to 239, image capture device 120should remain idle to ensure that in each frame overlap region 270 issynched. To provide this delay in image capture device 120 to allowimage capture device 110 to acquire the image data outside of overlapregion 270, image capture device 120 may be configured with a longervertical blanking period than image capture device 110.

It should be noted that in some embodiments as described above, thesynchronized portions of images from image capture devices 110 and 120correspond to overlap region 270. In other cases, the overlap portionsof image capture devices 110 and 120 may have one line synchronizedwithin overlap region 270, and moving away from that line, the lines maybecome less and less synchronized. This may be presented by dTL. Thesystem may reduce the rate at which the lines become unsynchronized by,for instance, adjusting the scan line rate of image capture device 120.For instance, where the focal length ratio between image capture devices110 and 120 is 2, the system could adjust TLn=0.5*TLw, which may resultin dTL=0. In another instance, TLn=0.6*TLw. In some embodiments,“synchronization” of image acquisition within overlap area 270 may beachieved even if the timing of image acquisition in that region by bothimage capture devices is not identical. For example, adequatesynchronization (e.g., to enable stereo analysis) may be achieved if adifference in timing between acquisition of the image portion fromdevice 120 in overlap region 270 is within 1%, 5%, 10%, or 15% of thetime required for device 110 to acquire the image portion correspondingto overlap region 270.

For example, as described above, where a particular ratio between focallengths exists between image capture devices 120 and 110 (e.g., 2:1),then a ratio of the line scan rate of the narrower FOV device 120 to theline scan rate of the wider FOV device 110 should be at least as high asthe focal length ratio in order to synchronize the acquired images overthe entire overlap region 270. In some embodiments, however, lower scanrate ratios may still be useful. For example, even with a line scan rateratio lower than the focal length ratio between image capture devices120 and 110, image portions from the device 110 and 120 images couldstill be synchronized over a portion (and often, a substantial portion)of overlap area 270.

Assuming a focal length ratio between devices 120 and 110 of 2:1, forexample, then a line scan ratio of at least 2:1 will enablesynchronization of images over the entire FOV overlap region 270.Synchronization of images over less than the entire region 270, however,may still be useful. For example, in some embodiments a line scan ratioof 1.9, 1.8. 1.7, 1.6, 1.5, or less, may still provide a useful level ofsynchronization between images from devices 110 and 120 in at least aportion of the area of overlap region 270.

The period of time during which acquisition of an image portion fromdevice 110 may be synchronized with acquisition of an image portion fromdevice 120 within overlap region 270 may be related to the ratio betweenthe line scan rate of device 110 and the line scan rate of device 120.For instance, where the ratio of FOVs is 2 but the maximum ratio of scanline timing is 1.8, the period of time may be based on the scan linetiming ratio of 1.8 (as well as on any applied pixel delays, horizontalblanking periods, etc.).

In addition to adjustment of the timing control parameters (e.g., linescan rate, vertical blanking period, horizontal blanking period, pixeldelay, etc.), other techniques may also be employed either together withor separate from adjustment of the timing control parameters. In someembodiments, image capture device 120 (the narrower field of viewdevice) may be configured with a lower resolution image sensor thanimage capture device 110. Thus, if image capture device 110 has an imagesensor with a resolution of 1280×960, then configuring image capturedevice 120 with an image sensor having a lower resolution (e.g.,640×480) may alleviate the need for a higher line scan rate in imagecapture device 120. Assuming the same 52 degree/26 degree exampledescribed above, a 1280×960 pixel sensor in image capture device 110could be synched with a 640×480 pixel sensor in image capture device 120within overlap region 270 even if both devices employed similar linescan rates. In this example, while image capture device 110 wasacquiring the 480 lines corresponding to overlap region 270 (i.e., lines240 to 720 in image capture device 110), image capture device 120 wouldbe acquiring its own 480 scan lines corresponding to overlap region 270.

Additionally, or alternatively, the demand for a higher scan rate in thenarrower field of view device may be alleviated if the narrower field ofview image capture device did not sample all image lines in each frame.For example, during acquisition of one frame, image capture device 120may be configured such that it samples only a subset of the availablescan lines (e.g., the odd numbered scan lines). During capture of thenext frame, image capture devices 120 may be configured such that itsamples the even numbered scan lines. Through this type of interlacingtechnique, the amount of time required to acquire the image dataassociated with overlap region 270 may be halved in comparison to atechnique in Which all scan lines are acquired for every frame.

In another example relative to selection of the timing controlparameters, if a line scan rate of image capture device 120 may beincreased, then the target ratio between focal lengths may also bereduced so that these match the ratio between line timing. For exampleif image capture device 110 has a horizontal FOV 210 of 46° and isrunning at a line timing of 45 KHz (22.2 usec) but the sensor cansupport line timing of 60 KHz (15.2 usec) then a 32° horizontal FOV 230of image capture device 120 can be supported and synchronized:

$\begin{matrix}{\frac{15.2_{usec} \times 46{^\circ}}{22.2_{usec}} = {31.7{{^\circ}.}}} & (3)\end{matrix}$

Here, the TLn may be 15.2 usec TLw may be 22.2 usec, and the widehorizontal FOV 210 may be 46°.

An image capture device 120 with a 30° horizontal FOV 230 mightrepresent a suitable solution for providing increased resolution (by50%) and small value of δTL:δT _(L) =αT _(Ln) −T _(w)

Here

$\alpha = \frac{46}{30}$is the ratio between the two horizontal FOVs 210 and 230. For instance,the ratio between the wide horizontal FOV and the narrow horizontal FOVmay be 52/26; 25/28; 60/30; 60/32; 46/36; 46/23; or any other suitableratio. Such ratios may give the narrow FOV camera more range, and mayallow detection of smaller objects at a greater distance.

$\begin{matrix}{{\delta\; T_{L}} = {{{\alpha\; T_{Ln}} - T_{Lw}} = {{{\frac{46{^\circ}}{30} \times 15.2_{usec}} - 22.2_{usec}} = 1.1_{usec}}}} & (5)\end{matrix}$

Any suitable ratio of focal lengths/fields of view may be selectedrelative to image capture device 110 and image capture device 120. Insome embodiments, this ratio may be set at 1.5, 1.6, 1.7, 1.8, up to 2,or greater than 2.

As noted above, both image capture device 110 and image capture device120 may be configured to output frames at the same rate. For example,the total number of clocks per frame (including acquired scan lines andblanking periods, delays, etc.) may be equal in both devices. And, toachieve the desired synch in overlap region 270, the line scan ratetiming multiplied by the number of rows covering overlap region 270 maybe the same for the two image capture devices.

Once the cameras have been synched, they may remain in lock-stepoperation such that the desired synch in overlap region 270 and thesimilar frame rates are preserved. The system may include one or moreprocessing devices (e.g., processing device 130) that may periodicallyperform a timing check to determine whether the first image capturedevice 110 and the second image capture device 120 remain synchronizedin capturing image data Within overlap region 270 and/or in overallframe rate. Processing device 130 may adjust one or more timingparameters or may reset one or more aspects of the image scan (e.g., ahard jump to a desired scan line number, such as the first scan line, orto any other scan line within the image frame) if a lack ofsynchronization is observed. Such a resynchronization process may occurperiodically throughout operation of image capture devices 110 and 120.

Image capture devices 110 and 120 may be used for independentapplications in some embodiments. In other embodiments, synching of theimage capture in overlap region 270, for example, may enableapplications that combine the information from both devices to providestereo analysis (e.g., stereo depth determination, among others).

In addition to using rolling shutters in image capture devices 110 and120, as described above, some embodiments may employ a global shutter.In such embodiments, rather than capturing image frames as a series ofsequentially acquired image scan lines, all pixel rows in a particularimage from are sampled and acquired at the same time.

FIG. 3 represents an exemplary process for use in systems employingimage capture devices with global shutters. As shown in FIG. 3, for asystem where the image capture devices 110 and 120 have global shutters,stereo processing in the overlap region 270 may involve simply croppingthe wide FOV camera image data outside of overlap region 270 (step 310).Next, the image data of the narrow FOV image capture device may besmoothed and subsampled (step 320) in order to get a matching imagepair. Stereo processing may then be completed based on the matchingimage pair (step 331). If a matching image pair is not obtained, arectification step might be required in practice (step 332).

In the rolling shutter embodiments described above, timing controlparameters etc. may be controlled in order to synchronize capture ofimage data within overlap region 270 and also to synchronize overallframe rates between image capture devices 110 and 120. In addition tothis front-end, hardware-based synchronization solution, otherembodiments may rely upon back-end processing as a basis for the stereoanalysis of image data from overlapping region 270. That is, rather thanforcing image capture device 110 and 120 to acquire image data in theoverlapping region 270 over the same time period, this image data can becaptured at different time periods, and processing may be performed onthe captured image data in order to enable meaningful stereo analysisbased on this data.

The back-end processing solutions may rely on the fact that a singlescan line of image capture device 110 may be synchronized with a singlescan line of image capture device 120. The particular lines in eachimage capture device that are synchronized may be selectable such thatin different image frames, the selected synchronized line may change.

In a particular image frame, the image data collected from both imagecapture devices 110 and 120 will correspond to one another at theselected synchronized scan line. In image capture devices that havedifferent fields of view, such as those shown in FIGS. 2a and 2b , scanlines away from the synchronized line will be out of synch. Further theloss of synchronization in image scan lines increases away from theselected synchronized line.

Where scan lines lack synchronization, disparities in the image data mayexist as a result of corresponding scan lines (e.g., in overlap region270) being captured during different time periods. For example, if anobject is moving through a scene relative to image capture devices 110and 120, then at the synchronized line, the captured image data fromboth of image capture devices 110 and 120 will indicate the sameposition for the object (because the synchronized scan line in eachimage capture device is captured during the same time period). Imagescan lines acquired away from the synchronized line, however, may lacksynchronization and, therefore, may result in position disparities forthe moving object.

Returning to the example of an image capture device 110 with a FOV of 52degrees and an image capture device 120 having a FOV of 26, where bothdevices include image sensors with a resolution of 1280×960, theselected synchronized line may occur at any selected line location. Insome embodiments, the synchronized line may occur within overlap region270. For example, the synchronized scan line may be selected at a centerof overlap region 270. Thus, while the image data acquired at thesynchronized line at the center of overlap region 270 will match intime, the image data captured at lines away from the synchronized linewill not match in time. And, the level of timing mismatch increases awayfrom the synchronized line. In the example where the central scan linein both image capture devices is synchronized, then the first scan lineof overlap region 270 will be captured at different times by imagecapture devices 110 and 120. For example, in image capture device 120(the narrower FOV device), the first scan line acquired in overlapregion 270 will correspond to the first scan line acquired in the frame.On the other hand, the first scan line acquired in overlap region 270 byimage capture device 110 will occur at scan line number 240. But, at thesame time image capture device 110 is acquiring scan line 240, imagecapture device 120 may also be acquiring its own scan line 240. In imagecapture device 120, however, scan line 240 is already 25% throughoverlap region 270. Thus, it can be seen that in image capture device110, the first scan line acquired in overlap region 270 will be acquiredlater than the first image scan line in overlap region 270 is capturedby image capture device 120.

This time difference, which decreases approaching the selectedsynchronized line, can result in image disparities. For example, in asimple example, a moving object may appear in different locations toimage capture device 110 than it does to image capture device 120 inscan lines away from the synchronized line. The extent of the apparentposition difference may increase as the timing difference increases awayfrom the synchronized line.

The extent of the introduced disparities, however, is predictablebecause the line scan timing is known in both image capture device 110and image capture device 120. Thus, processing device 130 may analyzethe acquired image data from both devices and may be able to obtainstereo analysis information from overlap region 270 by taking intoaccount the expected disparities introduced by the line scanning timingdifferences.

Because the line scan timing converges at the selected synchronizedline, there may exist a strip of scan lines about the synchronized linein which the introduced disparities are small or even negligible. Thus,in a strip of scan lines around the synchronized scan line, lessprocessing may be required to account for observed disparities. In someembodiments, this low disparity strip may be centered about an object ofinterest by judiciously selecting the synchronized scan line to overlapwith the object of interest.

Different types of objects encountered on a roadway (e.g., uprightobjects, moving objects, low objects near to the road surface, etc.) maybehave differently in terms of the acquired image disparities. Thus,selection of the synchronized scan line may depend on the type of anobject of interest in a scene (e.g., within overlap region 270).

As noted above, the synchronized line location may be adjusted.Synchronizing a particular line may be performed on any image capturedevice 110, 120 which provides basic synchronization technology,potentially allowing any particular row to be chosen. However, if onlyone line can be synchronized, there may not be one optimal line for allsituations. That is, it might be useful to switch the line that issynchronized between two or more image frames. Such an adjustment of theselected synchronized scan line may be accomplished by any suitabletechnique.

For example, in some embodiments, the vertical blanking period of thenarrow FOV image capture device 120 may be adjusted between frames. Insome embodiments, the vertical blanking period may be selected such thatimage acquisition by image capture device 120 falls behind and thencatches up with image capture device 110 such that different lines aresynchronized in different image frames. In another embodiment, thevertical blanking of the narrow FOV image capture device 120 could beset at a desired difference relative to image capture device 110. Forexample, if the vertical blanking period of image capture device 120 is100 lines fewer than the wide FOV image capture device 110, then inevery subsequent frame, the synchronized line location will vary by acertain amount (e.g., y=−100).

In one instance, the system may be configured so that selection of thesynchronized line depends on a speed of the vehicle. In anotherinstance, the system may be configured such that the selected first scanline and the selected second scan line both correspond to apredetermined distance away from the vehicle (e.g., where an object ofinterest may be located).

As noted above, where only a single scan line is both synchronized andspatially aligned between image capture devices 110 and 120, the linesabove and below this synchronized line may gradually become less andless in synch as the distance from the synchronized line increases:δt(y)=(y−y ₀)δT _(L)  (1)

where y₀ may be the one line in both images that is correctlysynchronized and δTL is:δT _(L)=2T _(Ln) −T _(Lw)  (2)

where TLn may be the line timing of the narrow camera 120 and TLw may bethe line timing of the wide camera 110. Often TLn=TLw and then δTL maybe the timing of one line (e.g. 1/45,000 sec).

Further, as noted above, if only one line can be synchronized, that linemay be chosen based on the requirements of a particular application. Forinstance, a synchronized line around y=−100 might be advantageous forroad surface applications. A row closer to the horizon might be usefulfor general object detection. Thus, it might be useful to switch theline than is synchronized between two or three (or more) settings, forexample: between row y=0 and row y=−100.

In a specific example, assuming two imagers are synchronized to thefirst imager row, in this case, the center line of the common image(y=0) may be synchronized. If the vertical blanking period of narrow FOVdevice 120 is 100 lines longer than the wide FOV device 110, in the nextframe it will be line y=−100 that may be synchronized. If no changes aremade then in the following frame line y=−200 may be synchronized and soforth. This may eventually loop around and with care the line y=0 mayagain come into synch. If, for example, there are 1000 lines and thevertical blanking is 500 lines then this may happen after 15 frames.

In another instance, the system may have the vertical blanking of thenarrow FOV device 120 set to 100 lines fewer than the wide FOV device110 (for instance, 400 versus 500 lines). This may not be changed butevery second frame of the narrow FOV camera 120 would be resynchronizedto the wide FOV camera 110. The wide FOV camera 110 may be free running(aka master). In the Aptina 1M Pixel sensor, synchronization may be doneusing the trigger pin. The trigger pin may be set to ‘0’ after thereadout of the second frame (for instance, where y=−100 issynchronized). The trigger pin may be set to ‘1’, N lines prior toreadout start from the wide FOV camera 110. N may be determined by thelength of integration time set in the narrow FOV camera 120. If N issmall, there may be time during the vertical blanking to lower and raisethe trigger pin. In night conditions and very long integration times,the narrow FOV camera 120 may have to skip a frame.

In one embodiment, the system may be configured so that correlation ofthe at least a first area of the first overlap portion of the firstimage with a corresponding second area of the second overlap portion ofthe second image includes selection of a first scan line associated withacquisition of the first image to be synchronized with a second scanline associated with acquisition of the second image; and performance ofa dense depth map computation in a horizontal strip within the firstoverlap portion of the first image and within a corresponding region ofthe second overlap portion of the second image. The system may beconfigured such that the horizontal strip represents at least 100 pixelsin width.

The system may then process different strips of the images for eachsetting. If processing the whole image every frame might be toocomputationally expensive, processing optimized strips might represent aviable alternative. In one instance, different strips of the images maythen be processed for each setting of image capture devices 110, 120.

The system may be configured such that correlation of the at least afirst area of the first overlap portion of the first image with acorresponding second area of the second overlap portion of the secondimage includes computation of dense optical flow between the firstoverlap portion of the first image and the second overlap portion of thesecond image; and use of the dense optical flow to warp at least one ofthe first or second images. The system may process the acquired imagesin order to match object features in the first image against those inthe second image in order to establish correspondences. The system maycreate a disparity map containing depth information for each pixel.Specifically, the disparity map computation may involve combination ofinformation from the imagers 110 and 120. The disparity may representthe distance, typically in pixels, between the x-coordinates ofcorresponding points in the first image and the second image. For eachpixel in the first and/or second image, the disparity map may containthe disparity and the depth information for the image point. The systemmay generate disparity images at different resolutions.

In one instance, the system ay determine depth maps for image capturedevices 110 and 120 to obtain depth information relative to objectswithin a scene. The depth map generated using disparity images at oneresolution may be different than those generated from the disparityimages at a different resolution. The depth map may be a two-dimensionalarray of pixels with each pixel value indicating the depth to a point inthe scene.

The system may be configured such that the at least one frame from thefirst plurality of images is warped to achieve synchronization with atleast one frame of the second plurality of images. The system may beconfigured such that the at least one processing device is furtherconfigured to preprocess the second image by a homography of a targetplane at a predetermined location.

The system may be configured such that correlation of the at least afirst area of the first overlap portion of the first image with acorresponding second area of the second overlap portion of the secondimage includes prewarping at least one of the first image or the secondimage according to a speed dependent function, the prewarping beingachieved by computing a homography matrix associated with motion of thevehicle; and computing a homography associated with skew of lines of atleast one of the first image or the second image.

In another embodiment, a solution may be to compute dense optical flow(u, v) between consecutive frames of one of the cameras and then usethis flow and warp one of the frames so that each line is what would beexpected if it was truly synchronized with the other camera:

$\begin{matrix}{{u(y)} = {u\frac{{yT}_{L}}{T_{F}}}} & (6) \\{{v(y)} = {v\frac{{yT}_{L}}{T_{F}}}} & (7)\end{matrix}$

One embodiment of the system may relate to depth measurements to uprightobjects, for instance, and may include choosing a line to synchronizeand then performing dense depth map computation in a horizontal stripsuch as ±100 pixels around the line. The selected line may be dependenton vehicle speed. The selected line may also depend on the location of aspecific target. The narrow FOV image may be preprocessed by ahomography of a target plane to increase the minimum operating depthwhere the search can be done along the horizontal line. When matchinglarge disparities, corresponding to closer targets, adjacent lines maybe searched to determine the matches. The focus of expansion (FOE) maybe estimated so as to further allow for small divergence of the epipolarlines. If the FOE is known, it may be possible to solve for distanceestimates taking into account the time lag. Where the vehicle movespredominantly in the forward direction, small errors in the FOE mayallow for significant correction. Errors may be reduced to below 1 m inalmost the whole range. Vehicle rotation may be measured from the imagesor from inertial sensors and may be modeled accurately.

In one instance, the system may have a stereo baseline of B=0.06 m, afocal length of wide camera 110 of f=1600 pixels (e.g., a camera with a6 mm lens and 3.75 um pixel) and a focal length of the narrow camera 120of 3200 pixels (e.g., 12 mm lens and same pixel). In one configuration,the stereo baseline may be B=dx and dh=dz=0. The narrow FOV image may becaptured full frame and subsampled in software reducing the effectivefocal length to the same 1600 pixel. The wide camera 110 line timing maybe: TLw= 1/45 msec which may be as fast as the sensor can provide andmay produce the minimum time lag inside the frame. In that case, thesystem could do TLn=TLw.

For a static target at a distance of Z1 the expected disparity may be:d=f*B/Z ₁  (8)along the epipolar lines. Assuming the images were rectified these linesmay be the image rows and matching may be a 1 dimensional search. If thehost vehicle is moving toward the target at a speed of V the imagepoints may move away from the focus of expansion (FOE) at a rate of:

$\begin{matrix}{{dx}^{\prime} = {\left( {x_{1} - x_{ep}} \right)\frac{dZ}{Z_{1}}}} & (9) \\{{dy}^{\prime} = {\left( {y_{1} - y_{ep}} \right)\frac{dZ}{Z_{1}}}} & (10)\end{matrix}$where (xep, yep) may be the FOE. However dZ may depend on the image row:dZ(y)=δt(y)V=(y ₂ −y ₁)δT _(L) V  (11)

The corresponding point (x2, y2) in the second image may now be givenas:

$\begin{matrix}{x_{2} = {d + {dx}^{\prime} + x_{1}}} & {{~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~}(12)} \\{= {{f*{B/Z_{1}}} + {\left( {x_{1} - x_{ep}} \right)\frac{\left( {y_{2} - y_{1}} \right)\delta\; T_{L}V}{Z_{1}}} + x_{1}}} & {(13)}\end{matrix}$ and $\begin{matrix}{y_{2} = {{dy}^{\prime} + y_{1}}} & {{~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~}(14)} \\{= {{\left( {y_{1} - y_{ep}} \right)\frac{\left( {y_{2} - y_{1}} \right)\delta\; T_{L}V}{Z_{1}}} + y_{1}}} & {(15)}\end{matrix}$

Here, one may assume yep=0, even though yep may be nonzero. Equation 15may be solved for y2 as a function of y1:

$\begin{matrix}{y_{2} = \frac{\frac{y_{1}^{2}\delta\; T_{L}V}{Z}}{1 + \frac{2_{y\; 1}\delta\; T_{L}V}{Z}}} & (16)\end{matrix}$

Due to the distance dependent shift in y, finding correspondences may bemore difficult than in the global shutter or static case. Oncecorrespondences are found then the errors in disparity caused by theshift in x may cause significant errors in distance estimates. Theseproblems may be mitigated as discussed in the various embodiments of thepresent system.

FIG. 4 is an example of displacement of points in the image from thesynchronized line for targets at various distances from the vehicle. Inone instance. FIG. 4 shows y displacement for targets at variousdistance values from the vehicle assuming V=30 m/s.

For instance, as shown in FIG. 4, the FOE may be at (xep, yep) of (0,0).Results are shown for y1 values between −250 and 250 pixels for targetsat different distances from the vehicle Z, for instance, Z=2.5 m (410),Z=5 m (420), Z=10 m (430), Z=20 m (440) and Z=30 m (450). For each rowthere may be a fixed displacement in y2 that may depend on V and Z andmay be almost quadratic in y1 for small values of:

$\frac{Z_{y\; 1}\sigma\; T_{L}V}{Z}.$The shift in y2 may be positive (or negative, depending on the directionof the rolling shutter). It may not be zero mean.

In one instance, the value y1, which may represent the y coordinate ofthe target in the first image, may be given. The y coordinate of thecorresponding target in the second image may be represented by y2. In awell synchronized and rectified system, y2 may be equal to y1. Thecorresponding target in the second image may be on the same row.However, due to the forward motion of the image capture devices, thefirst image and the second image may flow radially out from the FOE. Dueto the time difference, the target in the first image may have movedmore than the corresponding target in the second image. The outward flowin the first image and the second image may be different and may resultin y2 being different from y1.

For targets at a distance of over 10 m, the y2 displacement in theregion ±100 pixels from the synchronized line y0 may be less than 1pixel and search along the epipolar line may work. Beyond that regionthe displacement may be large and the search may be two dimensional andmay depend on the FOE. For instance, for targets at a distance ofapproximately 5 m, the y2 displacement in the region ±100 pixels fromthe synchronized line y0 may be close to 1.5 pixels. In anotherinstance, for targets at a distance of approximately 2.5 m, the y2displacement in the region ±100 pixels from the synchronized line y0 maybe close to 2.5-3 pixels.

FIG. 5 is an example of depth estimates for a target at a given distancefrom a vehicle. In one instance, FIG. 5 may show depth estimates for atarget at 10 m for vehicle forward motion of 30 m/s. Image x positionmay be translated into lateral distance using the known depth. There maybe a significant error in the disparity which similarly depends on row,vehicle velocity and target distance.

In another instance according to FIG. 5, results are shown forhorizontal rows between −100 and 100 around the synchronized row (every20 pixels). For instance, results are shown for horizontal row −100pixels (510) from the synchronized row; horizontal row 100 pixels (592)from the synchronized row; horizontal row −80 pixels (520) from thesynchronized row; horizontal row 80 pixels (591) from the synchronizedrow; horizontal row −60 pixels (530) from the synchronized row;horizontal row 60 pixels (590) from the synchronized row; horizontal row−40 pixels (540) from the synchronized row; horizontal row 40 pixels(580) from the synchronized row; horizontal row −20 pixels (550) fromthe synchronized row; horizontal row 20 pixels (570) from thesynchronized row; and horizontal row 0 pixels (560).

FIG. 6 shows displacement in y for various distance values after a fixinvolving pre-processing an image with the homography of a target plane.In one instance, FIG. 6 shows displacement in y for various distancevalues assuming V=30 m/s after correcting for plane at 10 m. The shiftin y may be approximated by a homography. If the narrow FOV image ispreprocessed by a homography for some intermediate distance such as 10m, then the minimum distance for which the y displacement is less than±1 pixel may be reduced and/or improved to 5 m.

For instance, for targets at a distance of over 10 m, they displacementin the region ±100 pixels from the synchronized line y0 may be less than−1 pixel. For targets at a distance of approximately 5 m, the ydisplacement in the region ±100 pixels from the synchronized line y0 maybe less than 1 pixel. In another instance, for targets at a distance ofapproximately 2.5 m, they displacement in the region ±100 pixels fromthe synchronized line y0 may be close to 2 pixels.

FIG. 7 shows exemplary process for use in systems to perform depthmeasurements for upright objects In another embodiment, the system maychoose an optimal line to synchronize (step 710).

The narrow FOV image may be preprocessed by a homography of a targetplane at 10 m (step 720). This may increase the minimum operating depthwhere the search can be done along the horizontal line.

The system may then perform dense depth map computation in a horizontalstrip such as ±100 pixels around that synchronized line (step 730). Forexample, one might synchronize image lines corresponding to a line onthe road 30 m ahead.

The optimal line might be speed dependent where the system maysynchronize lines corresponding to a line on the road 2 seconds ahead.The optimal line might also depend on the location of a specific target.

When matching larger disparities (step 740), corresponding to closertargets, the system may search for matches in adjacent lines (step 741).

The FOE may be estimated so as to further allow for the small divergenceof the epipolar lines (step 750). The process may not be very sensitiveto small errors in detecting the epipole.

If the FOE is known it may be possible to solve equation 13 for Z takingin to account the time lag (step 751). Since the vehicle motion may bepredominantly in the forward direction, small errors in the FOE, or eventaking xep=0, may allow for significant correction.

FIG. 8a shows depth estimates for a target at a given distance from thevehicle for a given vehicle forward motion and lateral velocity. Forinstance, FIG. 8a may show an example of depth estimates for a target at10 m for vehicle forward motion of 30 m/s and lateral velocity of Vx=2m/s.

In an instance according to FIG. 8a , results are shown for horizontalrows between −100 and 100 around the synchronized row (every 20 pixels).For instance, results are shown for horizontal row −100 pixels (810)from the synchronized row; horizontal row 100 pixels (892) from thesynchronized row; horizontal row −80 pixels (820) from the synchronizedrow; horizontal row 80 pixels (891) from the synchronized row;horizontal row −60 pixels (830) from the synchronized row; horizontalrow 60 pixels (890) from the synchronized row; horizontal row −40 pixels(840) from the synchronized row; horizontal row 40 pixels (880) from thesynchronized row; horizontal row −20 pixels (850) from the synchronizedrow; horizontal row 20 pixels (870) from the synchronized row; andhorizontal row 0 pixels (860).

FIG. 8b shows corrected depth estimates for forward vehicle motion andno lateral velocity. In one instance, FIG. 8b may show depth estimatescorrected assuming forward motion only (for instance, assuming Vx=0).Errors may be reduced to below 1 m in almost the whole range.

In an instance according to FIG. 8b , results are shown for horizontalrows between −100 and 100 around the synchronized row (every 20 pixels).For instance, results are shown for horizontal row −100 pixels (811)from the synchronized row; horizontal row 100 pixels (896) from thesynchronized row; horizontal row −80 pixels (821) from the synchronizedrow; horizontal row 80 pixels (895) from the synchronized row;horizontal row −60 pixels (831) from the synchronized row; horizontalrow 60 pixels (894) from the synchronized row; horizontal row −40 pixels(841) from the synchronized row; horizontal row 40 pixels (881) from thesynchronized row; horizontal row −20 pixels (851) from the synchronizedrow; horizontal row 20 pixels (871) from the synchronized row; andhorizontal row 0 pixels (861).

In another aspect, vehicle rotation may be measured from the images orfrom inertial sensors and may be modeled accurately. The narrow FOVimage, for example, may then be corrected. Yaw may cause horizontalskew. Pitch may cause vertical compression or elongation around thesynchronized line.

Another embodiment of the system may relate to estimation of distancesto road features and may include prewarping the narrow FOV imageaccording to a speed dependent function. A more general approach mayinvolve (1) computing the homography matrix due to the motion of thevehicle over the road plane per line timing; (2) for each line,computing homography for that line's skew; and (3) warping that lineaccording to the line's homography. Each line's homography may beapproximated by (1) computing a homography matrix for the timedifference between one frame and the next, (2) computing the imagemotion due to the homography and (3) for each line, warping with afraction of that motion depending on the time skew. The homographymatrix may also be estimated directly from two consecutive narrow FOVimages. A correction may be needed and may be performed as describedother embodiments. Improved performance on the road surface estimationmay be obtained by optimizing the image row which is synchronized.

In one instance, the two cameras may be mounted near the rearview mirrorat 1.25 m above the road surface. The vehicle speed may be 30 m/s. Dueto the forward motion of the car, points on the road in the image mayexperience a downward and outward flow. The magnitude of the flow mayincrease with the time difference and inversely with the distance to theroad surface at that point. The latter distance may be inverselyproportional to the image row:

$\begin{matrix}{Z = \frac{- {Hf}}{y}} & (17)\end{matrix}$

FIG. 9 shows the disparity expected on the road surface due to thecamera baseline with no time delay and with time delay. In particular,FIG. 9 shows a simulation of stereo disparity for a road surface:original points in wide angle camera, matching coordinates in narrowcamera assuming static scene and actual location due to time delay andforward motion. The time delay may increase linearly from thesynchronized line y=0.

FIG. 10 shows the shift in they direction, due to time delay and forwardmotion, as a function of the image row (curve 1010). Curve 1010 may be acubic function. The y shift may be very small (under 0.5 pixels) down torow y=−100 which may be 19 m ahead. Below row y−100, the system may needto make some adjustment in the search space

For instance, between row 0 to row −124, the system may search for thetarget centered on (x1,y1) along row (y2=y1); between row −125 and row−170, the system may search one row down (e.g. y2=y1−1); between row−170 and row −190, the system may search two rows down (e.g. y2=y1−2);between row −190 to −220, the system may search three rows down and soon.

In another instance, instead of switching the search row, the system maywarp the second image according to the function in FIG. 10. The warpingmay involve compression and/or rectification (e.g. rectification similarto the global shutter case) of the second image. To reduce the number ofwarping functions and the image blurring that may accompany warping, thetwo warps may be combined into a single mapping. The warping of thesecond image may give relatively smooth transitions.

FIG. 11 shows exemplary process for use in systems to estimate distancesto road features. For instance, in one solution, the processor 130 mayprewarp the narrow image vertically according to the speed dependent yshift function (step 410). There may be an advantage to prewarp assuminga specific distance such as 10 m.

A more general approach may be for the processor 130 to compute thehomography matrix H_(TT) due to the motion of the vehicle over the roadplane per line timing T_(L) (step 420):

$\begin{matrix}{{H_{\pi}\left( T_{L} \right)} = {{K\left( {R^{- 1} + \frac{\overset{\rightarrow}{T\;}{\overset{\rightarrow}{N}}^{\prime T}}{d_{\pi}^{\prime}}} \right)}K^{\prime - 1}}} & (18)\end{matrix}$

-   -   where R and T are the translation and rotation matrices for        motion in time T_(L) (taken for example from inertial sensors        and speedometer). In theory the plane normal N may also be        modified by R each line step. The approach may then involve the        processor 130 computing, for each line, homography for that        line's skew (step 430):        H _(π)(Y)=H _(π)(T _(L))^(Y)

The approach may then involve the processor 130 warping that lineaccording to homography for that line (step 441). The processor 130 mayapproximate the homography for a line by computing a homography matrixfor the time difference between one frame and the next (step 442),computing the image motion due to the homography (step 450) and then foreach line, warp with a fraction of that motion depending on the timeskew (step 460). The processor 130 may also estimate the homographymatrix directly from two consecutive narrow FOV images.

FIG. 12a shows the disparity error for different rows in the imageintroduced by the vehicle forward motion and the time delay. Inparticular, the figure shows the error in disparity due to time delayfor images rows starting at −20 (almost zero) to −240 (errors above 6pixels at image edges) every 20 pixels. For instance, results are shownfor horizontal row −20 pixels (1210); horizontal row −40 pixels (1220);horizontal row −60 pixels (1230); horizontal row −80 pixels (1240);horizontal row −100 pixels (1250); horizontal row −120 pixels (1260);horizontal row −140 pixels (1270); horizontal row −160 pixels (1280);horizontal row −180 pixels (1290); horizontal row −200 pixels (1291);horizontal row −220 pixels (1292); and horizontal row −240 pixels(1293).

FIG. 12b shows the disparity error as a fraction of the disparity onthat row (that is, true disparity). To get improved results, acorrection may be needed and may be performed to the disparities asdescribed in other embodiments. Improved performance on the road surfaceestimation may be obtained by optimizing the image row which may besynchronized. For instance, results are shown for horizontal row −20pixels (1211); horizontal row −40 pixels (1221); horizontal row −60pixels (1231); horizontal row −80 pixels (1241); horizontal row −100pixels (1251); horizontal row −120 pixels (1261); horizontal row −140pixels (1271); horizontal row −160 pixels (1281); horizontal row −180pixels (1295); horizontal row −200 pixels (1296); horizontal row −220pixels (1297); and horizontal row −240 pixels (1298).

FIG. 13 is an example of the expected y shift due to time delay andforward motion if a given row is synchronized. In one instance, FIG. 13shows the expected y shift due to time delay and forward motion if row−100 is synchronized (curve 1320) and the expected shift due to timedelay and forward motion if row 0 is synchronized (curve 1210). Row −100may correspond to a line on the row about 20 m in front of the car.Picking such a row may have advantages. First, the region where they-shift is below 0.5 pixels may increase down to −150. Secondly, giventhat row −100 may now be perfectly synchronized, the distance measure tothis row may be accurate and this may provide a good estimate of thevehicle pitch relative to the road plane. This may be used to improvethe prewarp in vertical direction and also to improve the x disparitycorrection.

In one instance, the image warp may be based on the equationZ=f*H/(y−y0). Where the horizon (y0) is assumed to be at line y=0,Z=f*H/y. If y0 is known, the system can solve for y′=y−y0 and then solvefor Z=f*H/y′, In other instances, y0 may not be known and may varybetween frames due to, for instance, vehicle pitch movement. In suchcases, y0 can be estimated using the distance Z100 at row −100 using thedisparity: (y100−y0)=f*H/Z100, or y0=y100−f*H/Z100. The adjusted y0 maygive a better correction warp, and therefore, a better x-disparitycorrection.

Another embodiment may relate to laterally moving objects or motiondisparity from any source and may include error reduction by usingmultiple frames. In general, the lateral disparity may be split into thetrue disparity as one would measure on a static scene and disparity dueto motion. The motion disparity may be approximately linear with thetime skew between the rows in the narrow and wide FOV cameras. Thelateral disparity at two different known times may provide two linearequations in the two unknowns (motion disparity and true disparity) andmay be solved to provide true disparity. The two samples may be obtainedbetween, for instance, a feature on an object viewed in the wide FOVimage and the same feature on the object viewed in the narrow FOV camerain two different frames. The two samples may also be obtained from twopoints on the object on different rows where we expect the same depthfor those points. A variant of this method may involve computing theoptical flow of a feature or dense flow between two images from the samecamera.

In one instance, an object moving laterally at a speed Vx may introducean error dvx to the static disparity due to baseline dB. The totaldisparity value may be equal to:

$\begin{matrix}{d = {{d_{B} + d_{V_{x}}} = {{\frac{fB}{Z} + \frac{\delta\;{t(y)}V_{x}}{Z}} = {\frac{fB}{Z} + \frac{{fy}\;\delta\; T_{L}V_{x}}{Z}}}}} & (20)\end{matrix}$

Depth accuracy may be related to the ratio:

$\begin{matrix}{\frac{d_{V_{x}}}{d_{B}} = \frac{y\;\delta\; T_{L}V_{x}}{B}} & (21)\end{matrix}$

The error may be constant along the row and linear in row distance fromthe synchronized row.

For validation of slow moving targets, the errors in disparity may notbe large. FIG. 14a is an example of the ratio of disparity error due tomotion over true disparity due to baseline as a function of row (y). Inone instance, FIG. 14a shows the ratio (of disparity error due to motionover true disparity due to baseline) as a function of row (y) for thecase where the object is moving laterally at 2 m/s (slow run) and abaseline of 0.06 m (curve 1410). The error ratio may be the expectederror in depth and may reach a maximum of about 15% at the top andbottom of the image. This might be considered acceptable for theapplication. In any case, a pedestrian may typically be in the centralregion where the error may be considerably less.

For the validation of fast moving targets, a method is described to giveaccurate results using multiple frames. FIG. 14b is an example of theratio of disparity error due to motion over true disparity due tobaseline for the case where the object is moving laterally. In oneinstance, FIG. 14b shows the ratio of disparity error due to motion overtrue disparity due to baseline for the case where the object is movinglaterally at 15 m/s (for instance, a car crossing at 50 kmh) (curve1420). The ratio may be above 20% in all but a narrow strip. The errormay be significantly reduced by using multiple frames.

FIG. 15 shows exemplary process for use in systems where there is motiondisparity. In general, the lateral disparity may be split into the truedisparity as one would measure on a static scene and disparity due tomotion, either of the host vehicle or the object (step 510). Thedisparity due to motion may be approximately linear with the time skewbetween the rows in the narrow FOV camera 120 and wide FOV camera 110(step 520). The lateral disparity at two different (known) times mayprovide two linear equations in the two unknowns, true disparity andmotion disparity, (step 530) and may be solved to provide the truedisparity (step 540). The two samples may be obtained between, forexample, a feature on an object viewed in the wide FOV image and thesame feature on the object viewed in the narrow FOV camera in twodifferent frames. The two samples may also be obtained from two pointson the object on different rows where we expect the same depth for thosepoints such as the edge of a vertical pole. A variant of this method maybe to compute the optical flow (u) of a feature or dense flow betweentwo images from the same camera. The motion disparity may then bedetermined by:

$\begin{matrix}{d_{V_{x}} = \frac{y\;\delta\; T_{L}u}{T_{F}}} & (22)\end{matrix}$

-   -   where TF is the time of one full frame including blanking. It        should be noted that this relay be approximately true not only        for laterally moving objects but motion disparity from any        source.

Moreover, while illustrative embodiments have been described herein, thescope of any and all embodiments having equivalent elements,modifications, omissions, combinations (e.g., of aspects across variousembodiments), adaptations and/or alterations as would be appreciated bythose skilled in the art based on the present disclosure. Thelimitations in the claims are to be interpreted broadly based on thelanguage employed in the claims and not limited to examples described inthe present specification or during the prosecution of the application.The examples are to be construed as non-exclusive. Furthermore, thesteps of the disclosed routines may be modified in any manner, includingby reordering steps and/or inserting or deleting steps. It is intended,therefore, that the specification and examples be considered asillustrative only, with a true scope and spirit being indicated by thefollowing claims and their full scope of equivalents.

What is claimed is:
 1. An imaging system, the system comprising: a first image capture device having a first field of view and configured to acquire a first series of image scan lines relative to a scene; a second image capture device having a second field of view different from the first field of view and that at least partially overlaps the first field of view, the second image capture device being configured to acquire a second series of image scan lines relative to the scene; and at least one processing device configured to identify at least one scan line from the first series of image scan lines and at least one scan line from the second series of image scan lines as a synchronized line and calculate a depth of at least one point based on the synchronized line.
 2. The imaging system of claim 1, wherein: the first image capture device has a first focal length; the second image capture device has a second focal length greater than the first focal length; and the at least one processing device is further configured to subsample the second series of image scan lines to reduce an effective focal length of the second image capture device to the first focal length.
 3. The imaging system of claim 1, wherein the at least one processing device is further configured to identify synchronized points in one or more lines of the first series of image scan lines and the second series of images scan lines adjacent to the synchronized line, and the calculated depth is further based on the synchronized points.
 4. The imaging system of claim 1, wherein the at least one processing device is further configured to calculate a disparity between points corresponding to the at least one point in the identified at least one scan line from the first series of images and the identified at least one scan line from the second series on images and to calculate the depth of the at least one point based on the disparity.
 5. The imaging system of claim 4, wherein calculating the depth based on the disparity is further based on a stereo baseline of the system.
 6. The imaging system of claim 1, wherein the at least one processing device is further configured to generate a depth map based on the calculated depth.
 7. The imaging system of claim 6, wherein the depth map is based on calculated depths of scan lines above and below the synchronized line.
 8. The imaging system of claim 1, wherein the at least one processing device is further configured to perform preprocessing on the first series of image scan lines and the second series of image scan lines based on homography of a target plane.
 9. The imaging system of claim 1, wherein the first image capture device has a first scan rate associated with acquisition of the first series of image scan lines that is different from a second scan rate associated with acquisition of the second series of image scan lines, such that the first image capture device acquires a first portion of the first series of image scan lines that overlaps a second portion of the second series of image scan lines over a period of time during which the second portion is acquired.
 10. The imaging system of claim 9, wherein the synchronized line is selected from a subset of scan lines from the second image capture device within the second portion.
 11. The imaging system of claim 1, the first series of image scan lines being captured using a first rolling shutter.
 12. The imaging system of claim 11, the second series of image scan lines being captured using a second rolling shutter.
 13. The imaging system of claim 12, wherein the at least one processing device is further configured to synchronize the first rolling shutter with the second rolling shutter.
 14. An imaging system, comprising: at least one processing device configured to: receive, from a first image capture device having a first field of view, a first series of image scan lines relative to a scene; receive, from a second image capture device having a second field of view different from the first field of view and that at least partially overlaps the first field of view, a second series of image scan lines relative to the scene; and correlate at least one scan line from the first series of image scan lines and at least one scan line from the second series of image scan lines as a synchronized line; and calculate a depth of at least one point based on the synchronized line.
 15. The imaging system of claim 14, wherein the at least one processing device is further configured to calculate a disparity between points corresponding to the at least one point in the identified at least one scan line from the first series of images and the identified at least one scan line from the second series on images and to calculate the depth of the at least one point based on the disparity.
 16. The imaging system of claim 15, wherein calculating the depth based on the disparity is further based on a stereo baseline of the system.
 17. The imaging system of claim 14, wherein the at least one processing device is further configured to identify synchronized points in one or more lines of the first series of image scan lines and the second series of images scan lines adjacent to the synchronized line, and the calculated depth is further based on the synchronized points.
 18. The imaging system of claim 14, wherein the at least one processing device is further configured to generate a depth map based on the calculated depth.
 19. The imaging system of claim 18, wherein the depth map is based on calculated depths of scan lines above and below the synchronized line.
 20. The imaging system of claim 14, wherein the at least one processing device is further configured to perform preprocessing on the first series of image scan lines and the second series of images scan lines based on homography of a target plane.
 21. The imaging system of claim 14, wherein the first image capture device has a first scan rate associated with acquisition of the first series of image scan lines that is different from a second scan rate associated with acquisition of the second series of image scan lines, such that the first image capture device acquires a first portion of the first series of image scan lines that overlaps a second portion of the second series of image scan lines over a period of time during which the second portion is acquired.
 22. The imaging system of claim 21, wherein the synchronized line is selected from a subset of scan lines from the second image capture device within the second portion.
 23. The imaging system of claim 14, the first series of image scan lines being captured using a first rolling shutter.
 24. The imaging system of claim 23, the second series of image scan lines being captured using a second rolling shutter.
 25. The imaging system of claim 24, wherein the at least one processing device is further configured to perform a periodic check to determine whether the first rolling shutter and the second rolling shutter are synchronized.
 26. The imaging system of claim 24, wherein the at least one processing device is further configured to perform resynchronization of the first rolling shutter and the second rolling shutter if the first rolling shutter and the second rolling shutter have fallen out of synch.
 27. The imaging system of claim 14, wherein the first image capture device has a first focal length, the second image capture device has a second focal length greater than the first focal length, and the at least one processing device is further configured to subsample the second series of image scan lines to reduce an effective focal length of the second image capture device to the first focal length.
 28. A method for depth computation, comprising: receiving, from a first image capture device having a first field of view, a first series of image scan lines relative to a scene; receiving, from a second image capture device having a second field of view different from the first field of view and that at least partially overlaps the first field of view, a second series of image scan lines relative to the scene; and correlating at least one scan line from the first series of image scan lines and at least one scan line from the second series of image scan lines as a synchronized line; and calculating a depth of at least one point based on the synchronized line. 